Ten-task TDD plan implementing the test bench spec: phase-timer hook,
fixture loader, token counter, single-fixture runner, Mode A
(history), --compare, Mode B (config cost), 8 fixtures, and README.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Bench harness needs to measure per-phase timing without duplicating the
pipeline loop. Default is a zero-cost no-op so normal hook invocations
are unaffected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

… test

Ruff UP037 flagged the self-referential return-type quotes on
_NoopPhaseTimer (the file doesn't use `from __future__ import annotations`,
so the annotations aren't actually needed on a private no-op class).
Also tighten the phase-timer test to assert the exact set of 7 phase
names, so a future refactor that drops one will fail the test.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

`bully bench` now dispatches to pipeline.bench. Flags parsed but not yet
implemented -- subsequent tasks fill in fixture loading, token counting,
and both modes. Also adds module-level __getattr__ to pipeline.py so
`from pipeline import bench` works in the flat-module test context.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Task 2's test attempted `from pipeline import bench`, which required a
module-level __getattr__ in pipeline.py to work with the test's sys.path
setup. That was the wrong fix -- the existing pattern for sibling-module
imports (see test_analyzer.py) is `import bench`, treating bench.py as a
top-level module once pipeline/ is on sys.path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Fixtures live in bench/fixtures/<name>/{config.yml,fixture.json}.
load_fixture validates both files exist and the metadata has required
fields; discover_fixtures returns all subdirectories sorted by name.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

count_tokens uses messages/count_tokens when an API key and the SDK are
available; otherwise returns len(json.dumps(payload)) + len(system) and
tags the result 'proxy'. System prompt is loaded from the real
bully-evaluator agent file so counts match production.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

PhaseTimer implements the pipeline hook protocol and records per-phase
elapsed time. run_fixture runs one fixture N times in-process, samples
cold-start once via subprocess, and builds the real semantic payload to
count tokens.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Previously, run_fixture called count_tokens even on fixtures with no
semantic rules -- producing a misleading non-zero token count for a
dispatch that would never happen. Fix to report 0 tokens with method
"n/a-no-semantic-rules" so history entries accurately reflect what real
bully invocations would send.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

\`bully bench\` now enumerates bench/fixtures/*, runs each fixture, and
appends one JSONL line to bench/history.jsonl with per-fixture phase
timings + aggregate totals. Human-readable summary to stdout by
default; --json emits the raw record.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Per-fixture + aggregate deltas for wall time and input tokens. Fails
clearly when history has fewer than two runs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

\`bully bench --config <path>\` parses a .bully.yml and reports floor
tokens per dispatch, per-rule marginal cost (sorted), diff scaling at
1/10/100/1000 added lines, and a scope-grouped breakdown. Script and
ast rules are listed separately as zero model-token cost.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Fixtures exercise: single-engine smoke (01-03), mixed-engine path (04),
multi-level extends chain (05), payload size scaling (06, 07), and the
auto-generated skip short-circuit (08). Integration test runs all of
them against run_mode_a.

Also fix bench.py import to work in both package context (pipeline.bench)
and direct sys.path context (used by pytest), and fix pre-existing ruff
isort violations in test_bench.py.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Covers Mode A (fixture suite + history.jsonl), Mode B (config cost
analysis), and the ANTHROPIC_API_KEY / optional anthropic SDK
requirement for real token counts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Previously run_fixture set BULLY_TRUST_ALL=1 permanently. Fine for the
CLI path (process exits) but leaks state if bench is consumed as a
library. Save the prior value and restore in a finally block.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Previously returned {returncode, report}; the dict wrapper was unique to
Mode B. Align with the simpler "None = failure, dict = success" shape
used elsewhere.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Previously `--config ... --compare` silently dropped --compare (config
branch won the if/elif). Use argparse's mutually_exclusive_group so
users get an explicit error.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Fixture targets (.py files under bench/fixtures/*) intentionally model
user code with unused vars, incomplete functions, and style violations
-- they are data, not source. Exclude them from ruff so CI stops
flagging them as real violations.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Bully can catch style violations at edit time before CI does. The
--force-exclude flag makes ruff respect pyproject's extend-exclude list
even when a single file is passed, so bench fixtures are skipped.

After pulling, run `bully trust --refresh` to re-approve the modified
config (trust gate hashes the .bully.yml contents).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

CI's `ruff format --check` step flagged formatting differences on both
bench files. Apply the formatter output directly; no semantic changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>